{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Load Data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "'''load these files through an API format\n",
    "ideally create some sort of input mechanism to extract specific subsections of the data'''\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Access Data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "'''access data files in some tabular, easy to access form\n",
    "recommended packages:\n",
    "globe_data_frame = SFrame (input)\n",
    "ice_data_frame = SFrame(input)'''\n",
    "import pandas\n",
    "import turicreate"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Feature Extraction"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'we need to extract the following features:\\n1. latitude\\n2. longitude\\n3. canopy elevation\\nlatitude / longitude will be our constraint inputs. Canopy elevation will be our desired output.\\nThis should look something like this:\\ndata_frame[lat][long] --> boolean condition --> some subset of tree heights within this range\\n'"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "'''we need to extract the following features:\n",
    "1. latitude\n",
    "2. longitude\n",
    "3. canopy elevation\n",
    "latitude / longitude will be our constraint inputs. Canopy elevation will be our desired output.\n",
    "This should look something like this:\n",
    "data_frame[lat][long] --> boolean condition --> some subset of tree heights within this range\n",
    "'''"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Clustering Algorithm"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\"we need to cluster the data into groups which:\\n1. are homogenous enough to be considered similar groupings\\n2. are distinct enough to be statistically distinguishable from other clusters of similar size\\n3. include enough Globe data points to be statistically significant\\n--> if condition 3 is not met, we must let the cluster = 0\\n\\nCandidates include:\\na) Qualitative: Group clusters by region\\nb) Quantitative, dynamic: Group clusters by average variance within a general region\\nc) Quantitative, fixed: Group clusters by positional data within a strict region\\nWe will likely perform some combination of b) and c), but we will keep a) in mind should our region size be so large \\nthat it mixes regions\\nWe have two choices as to how to approach this problem:\\n\\nQuantitative, dynamic:\\nRecommendation: k-means algorithm to sort Globe observations of similar sizes into groupings\\nPros: flexible --> doesn't rigidly force observations into a set geographic setting\\nCons: very complex --> difficult to return strict bounds for non-rigid regions\\na k-means algorithm could enable us to sort trees by biome / climate region / even accurate vs inaccurate schools,\\nas opposed to sticking with strict , pre-set regions\\n\\nQuantitative, fixed:\\nWe may want to make the sizes of the clusters fixed --> dynamically sized clusters may be out of the scope of this\\ninitial project.\\nProposal: 90 km x 90 km (~0.8 degrees lat x 0.8 degrees long) \\nPros: easy to work with / implement\\nCons: potentially has a higher algorithmic time complexity, O(t)\\n\\nRecommendation: break up chunks into 90 km x 90 km chunks; with extra time, we can employ k-means algorithm in order\\nto find inherent similarities / patterns within these chunks --> but we can always resort back to standardized chunks\\nBut, with a limited time frame, easiest to work with straightforward solution\\n\\n\\n\""
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "'''we need to cluster the data into groups which:\n",
    "1. are homogenous enough to be considered similar groupings\n",
    "2. are distinct enough to be statistically distinguishable from other clusters of similar size\n",
    "3. include enough Globe data points to be statistically significant\n",
    "--> if condition 3 is not met, we must let the cluster = 0\n",
    "\n",
    "Candidates include:\n",
    "a) Qualitative: Group clusters by region\n",
    "b) Quantitative, dynamic: Group clusters by average variance within a general region\n",
    "c) Quantitative, fixed: Group clusters by positional data within a strict region\n",
    "We will likely perform some combination of b) and c), but we will keep a) in mind should our region ]"
    "size be so large \n that it mixes regions\n",
    "We have two choices as to how to approach this problem:\n",
    "\n",
    "Quantitative, dynamic:\n",
    "Recommendation: k-means algorithm to sort Globe observations of similar sizes into groupings\n",
    "Pros: flexible --> doesn't rigidly force observations into a set geographic setting\n",
    "Cons: very complex --> difficult to return strict bounds for non-rigid regions\n",
    "a k-means algorithm could enable us to sort trees by biome / climate region / even accurate vs inaccurate schools,\n",
    "as opposed to sticking with strict , pre-set regions\n",
    "\n",
    "Quantitative, fixed:\n",
    "We may want to make the sizes of the clusters fixed --> dynamically sized clusters may be out of the scope of this\n",
    "initial project.\n",
    "Proposal: 90 km x 90 km (~0.8 degrees lat x 0.8 degrees long) \n",
    "Pros: easy to work with / implement\n",
    "Cons: potentially has a higher algorithmic time complexity, O(t)\n",
    "\n",
    "Recommendation: break up chunks into 90 km x 90 km chunks; with extra time, we can employ k-means algorithm in order\n",
    "to find inherent similarities / patterns within these chunks --> but we can always resort back to standardized chunks\n",
    "But, with a limited time frame, easiest to work with straightforward solution\n",
    "\n",
    "\n",
    "'''"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Sorting Algorithm utilizing mean normalization"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "'''\n",
    "After sorting the data into uniform clusters (per recommendation above),we will work w some \n",
    "n x m sized rectangles that will be:\n",
    "i) small enough to have a low variance within the cluster\n",
    "ii) large enough that our algorithm is not too inefficient / costly + ICE data sample is large enough to be\n",
    "statistically significant.\n",
    "\n",
    "We then want to place these clusters into a matrix. Their position in the matrix should be according to their\n",
    "position. If a lat / long position does not have a data cluster, we will mark that entry as 0 in the matrix.\n",
    "We will first create the Globe matrix. It will look something like this:\n",
    "\n",
    "[[cluster], [cluster], 0, ..., [cluster], 0 , [cluster]]\n",
    "Where each [cluster] = some collection of data points within that specific lat / long position\n",
    "and 0 represents that there is no cluster there.\n",
    "\n",
    "\n",
    "We then create an analagous Ice matrix. If there is a list representing a cluster in the Globe matrix at position A, \n",
    "we will fill in the mean value for the  datacluster for the Ice matrix at position A. If no globe data is present \n",
    "at position B, we will let entry = 0 for the ice matrix at position B.\n",
    "Why use mean?\n",
    "Given a cluster of ice data, we want to say: what does this cluster predict the approximate height should be?\n",
    "In a perfect scenario, we would use some hypothesis function.\n",
    "However, because:\n",
    "1) the region is a small area --> minimal deviation is fairly probable\n",
    "2) deriving a hypothesis function is a fairly costly process\n",
    "we will use the mean of the data as a substitute for the hypothesized value \n",
    "--> given a cluster of ice data points, we predict the Ice value will equal the mean of the cluster \n",
    "\n",
    "\n",
    "Ideally, the algorithm will resemble the following:\n",
    "input: subset of tree height clusters \n",
    "organize these Globe clusters into data structure\n",
    "organize ICE cluster means into data structure\n",
    "output: data structures of clusters vs mean / 'predicted value' ready for comparison\n",
    "\n",
    "**how to check if ICE data is homogenous (extra):\n",
    "normalize data:\n",
    "x(i) = [x(i) - x(mean)]/standard dev\n",
    "--> if mean of all x(i) values is close to zero, this means there is a low deviation, and the subset is good to go\n",
    "-->if the mean is large, this means there is significant dev, and we need to slice the cluster segments thinner\n",
    "we can make the segments of lat x long smaller and smaller until the mean of all x(i) approaches zero\n",
    "\n",
    "\n",
    "'''\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Compare Data Clusters"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'we now have two sets of data clusters. Our objective is to run these clusters through an algorithm that\\ncompares the data clusters and answers the question: is there a statistical difference between these clusters?\\nIf so, what is the error?\\n'"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "'''we now have two sets of data clusters. Our objective is to run these clusters through an algorithm that\n",
    "compares the data clusters and answers the question: is there a statistical difference between these clusters?\n",
    "If so, what is the error?\n",
    "'''"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Machine Learning Error Algorithm"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'given these two data structures, we want to to output an entry-wise comparison as follows:\\ngiven some ith, jth entry:\\nlet the globe cluster = [x(1), ..., x(i),...,x(n)]\\nlet the ice entry = theta\\nwe create a new value, lamda, equal to the following:\\nlambda = SUM[(theta - x(i))**2]/n\\nlambda = sqrt(lamdba)\\nthis will return a series lambda values which represent something called the cost function\\nThe cost function measures the following:\\nhow much deviation is there between the predicted value, theta, and the measured value, x(i)? how much deviation\\nis there over the set of all values of x(i)?\\nIn summary, the algorithm works as follows:\\ninput1: a list of globe observations over a given range\\ninput2: a predicted tree height value based on ICE data\\noutput: some error function , which represents the difference between predicted and measured.\\nThis output tells us: how much deviation is there between clusters?\\n\\nnow, to account for any systematic data (whether in the process itself or in measurements taken), we need to normalize\\nthe data. To do so, we follow the same steps as up above:\\ngiven this list of error values corresponding to a \\n\\n'"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "'''given these two data structures, we want to to output an entry-wise comparison as follows:\n",
    "given some ith, jth entry:\n",
    "let the globe cluster = [x(1), ..., x(i),...,x(n)]\n",
    "let the ice entry = theta\n",
    "we create a new value, lamda, equal to the following:\n",
    "lambda = SUM[(theta - x(i))**2]/n\n",
    "lambda = sqrt(lamdba)\n",
    "this will return a series lambda values which represent something called the cost function\n",
    "The cost function measures the following:\n",
    "how much deviation is there between the predicted value, theta, and the measured value, x(i)? how much deviation\n",
    "is there over the set of all values of x(i)?\n",
    "In summary, the algorithm works as follows:\n",
    "input1: a list of globe observations over a given range\n",
    "input2: a predicted tree height value based on ICE data\n",
    "output: some error function , which represents the difference between predicted and measured.\n",
    "This output tells us: how much deviation is there between clusters?\n",
    "To get an idea of the magnitude, we find the mean error --> this gives us an approximation for the \n",
    "magnitude of error for any given cluster\n",
    "'''"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Normalize Error"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "'''in order to account for any systematic error (whether in the process itself or in measurements taken), we also \n",
    "want to normalize the data. To do this, we follow the same algorithm that was employed previously:\n",
    "find the mean and standard deviation of the error list\n",
    "for each entry lambda in the list:\n",
    "lamda = (lambda - mean) / standard deviation\n",
    "these updated values of lambda should give a measure of distance from the mean error\n",
    "Therefore: we can expect a certain degree of error, which will be displayed in the error function. The error\n",
    "function will represent the magnitude of the error. \n",
    "This second list will provide insight into the relative magnitude and orientation of the error.\n",
    "Negative values --> very little error\n",
    "Zero Values --> very normal amount of error\n",
    "Non-negative values --> high amount of error\n",
    "We can produce a list of values and, corresponding to their position within the list, \n",
    "\n",
    "input: a list of error values corresponding to error at certain latitude, longitude positions\n",
    "output: a normalized list of error values, ranging from -1 to 1, corresponding to the MAGNITUDE\n",
    "of error at certain latitude, longitude\n",
    "\n",
    "\n",
    "\n",
    "'''"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
